{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 从外部源获取样本数据"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果条件允许，学本书内容时尽量用你熟悉的数据集；方便起见，我们用scikit-learn的内置数据库。这些内置数据库可用于测试不同的建模技术，如回归和分类。而且这些内置数据库都是非常著名的数据库。这对不同领域的学术论文的作者们来说是很用的，他们可以用这些内置数据库将他们的模型与其他模型进行比较。\n",
    "\n",
    ">推荐使用IPython来运行文中的指令。大内存很重要，这样可以让普通的命令正常运行。如果用IPython Notebook就更好了。如果你用Notebook，记得用`%matplotlib inline`指令，这样图象就会出现在Notebook里面，而不是一个新窗口里。\n",
    "\n",
    "<!-- TEASER_END -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##Getting ready"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "scikit-learn的内置数据库在`datasets`模块里。用如下命令导入："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn import datasets\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在IPython里面运行`datasets.*?`就会看到`datasets`模块的指令列表。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##How to do it…"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`datasets`模块主要有两种数据类型。较小的测试数据集在`sklearn`包里面，可以通过`datasets.load_*?`查看。较大的数据集可以根据需要下载。后者默认情况下不在`sklearn`包里面；但是，有时这些大数据集可以更好的测试模型和算法，因为比较复杂足以模拟现实情形。\n",
    "\n",
    "默认在`sklearn`包里面的数据集可以通过`datasets.load_*?`查看。另外一些数据集需要通过`datasets.fetch_*?`下载，这些数据集更大，没有被自动安装。经常用于测试那些解决实际问题的算法。\n",
    "\n",
    "首先，加载`boston`数据集看看："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Boston House Prices dataset\n",
      "\n",
      "Notes\n",
      "------\n",
      "Data Set Characteristics:  \n",
      "\n",
      "    :Number of Instances: 506 \n",
      "\n",
      "    :Number of Attributes: 13 numeric/categorical predictive\n",
      "    \n",
      "    :Median Value (attribute 14) is usually the target\n",
      "\n",
      "    :Attribute Information (in order):\n",
      "        - CRIM     per capita crime rate by town\n",
      "        - ZN       proportion of residential land zoned for lots over 25,000 sq.ft.\n",
      "        - INDUS    proportion of non-retail business acres per town\n",
      "        - CHAS     Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)\n",
      "        - NOX      nitric oxides concentration (parts per 10 million)\n",
      "        - RM       average number of rooms per dwelling\n",
      "        - AGE      proportion of owner-occupied units built prior to 1940\n",
      "        - DIS      weighted distances to five Boston employment centres\n",
      "        - RAD      index of accessibility to radial highways\n",
      "        - TAX      full-value property-tax rate per $10,000\n",
      "        - PTRATIO  pupil-teacher ratio by town\n",
      "        - B        1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town\n",
      "        - LSTAT    % lower status of the population\n",
      "        - MEDV     Median value of owner-occupied homes in $1000's\n",
      "\n",
      "    :Missing Attribute Values: None\n",
      "\n",
      "    :Creator: Harrison, D. and Rubinfeld, D.L.\n",
      "\n",
      "This is a copy of UCI ML housing dataset.\n",
      "http://archive.ics.uci.edu/ml/datasets/Housing\n",
      "\n",
      "\n",
      "This dataset was taken from the StatLib library which is maintained at Carnegie Mellon University.\n",
      "\n",
      "The Boston house-price data of Harrison, D. and Rubinfeld, D.L. 'Hedonic\n",
      "prices and the demand for clean air', J. Environ. Economics & Management,\n",
      "vol.5, 81-102, 1978.   Used in Belsley, Kuh & Welsch, 'Regression diagnostics\n",
      "...', Wiley, 1980.   N.B. Various transformations are used in the table on\n",
      "pages 244-261 of the latter.\n",
      "\n",
      "The Boston house-price data has been used in many machine learning papers that address regression\n",
      "problems.   \n",
      "     \n",
      "**References**\n",
      "\n",
      "   - Belsley, Kuh & Welsch, 'Regression diagnostics: Identifying Influential Data and Sources of Collinearity', Wiley, 1980. 244-261.\n",
      "   - Quinlan,R. (1993). Combining Instance-Based and Model-Based Learning. In Proceedings on the Tenth International Conference of Machine Learning, 236-243, University of Massachusetts, Amherst. Morgan Kaufmann.\n",
      "   - many more! (see http://archive.ics.uci.edu/ml/datasets/Housing)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "boston = datasets.load_boston()\n",
    "print(boston.DESCR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`DESCR`将列出数据集的一些概况。下面我们来下载一个数据集："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "downloading Cal. housing from http://lib.stat.cmu.edu/modules.php?op=modload&name=Downloads&file=index&req=getit&lid=83 to C:\\Users\\tj2\\scikit_learn_data\n",
      "California housing dataset.\n",
      "\n",
      "The original database is available from StatLib\n",
      "\n",
      "    http://lib.stat.cmu.edu/\n",
      "\n",
      "The data contains 20,640 observations on 9 variables.\n",
      "\n",
      "This dataset contains the average house value as target variable\n",
      "and the following input variables (features): average income,\n",
      "housing average age, average rooms, average bedrooms, population,\n",
      "average occupation, latitude, and longitude in that order.\n",
      "\n",
      "References\n",
      "----------\n",
      "\n",
      "Pace, R. Kelley and Ronald Barry, Sparse Spatial Autoregressions,\n",
      "Statistics and Probability Letters, 33 (1997) 291-297.\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "housing = datasets.fetch_california_housing()\n",
    "print(housing.DESCR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##How it works…"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "当这些数据集被加载时，它们并不是直接转换成Numpy数组。它们是`Bunch`类型。**Bunch**是Python常用的数据结构。基本可以看成是一个词典，它的键被实例对象作为属性使用。\n",
    "\n",
    "用`data`属性连接数据中包含自变量的Numpy数组，用`target`属性连接数据中的因变量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X, y = boston.data, boston.target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "网络上`Bunch`对象有不同的实现；自己写一个也不难。scikit-learn用[基本模块](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/datasets/base.py)定义`Bunch`。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##There's more…"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "让你从外部源获取数据集时，它默认会被当前文件夹的`scikit_learn_data/`放在里面，可以通过两种方式进行配置：\n",
    "\n",
    "- 设置`SCIKIT_LEARN_DATA`环境变量指定下载位置\n",
    "- `fetch_*?`方法的第一个参数是`data_home`，可以知道下载位置\n",
    "\n",
    "通过`datasets.get_data_home()`很容易检查默认下载位置。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##See also"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "UCI机器学习库（UCI Machine Learning Repository）是找简单数据集的好地方。很多scikit-learn的数据集都在那里，那里还有更多的数据集。其他数据源还是著名的KDD和Kaggle。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
